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ABSTRACT 

Rollover  incidents  of  military  vehicles  have 
resulted  in  soldiers  incurring  injuries  or  losing  their 
lives.  A  recent  report  identified  that  one  cause  of 
vehicle  rollovers  is  the  driver’s  inability  to  assess 
rollover  threat,  such  as  a  cliff,  soft  ground,  water,  or 
culvert  on  the  passenger  side  of  the  vehicle.  The 
vehicle’s  width  hinders  the  driver’s  field  of  view.  To 
reduce  the  number  of  military  vehicles  rolling  over,  a 
road  edge  detection  and  driver  warning  system  is 
being  developed  to  warn  the  driver  of  potential 
rollover  threats  and  keep  the  driver  from  veering  off 
the  side  of  the  road.  This  system  utilizes  a  unique, 
ultra-fast,  image-processing  algorithm  based  on  the 
neurobiology  of  insect  vision,  specifically  fly  vision. 
The  system  consists  of  a  Long-Wavelength  Infrared 
(LWIR)  camera  and  monochrome  video  camera 
system,  a  long-range  laser  scanner,  a  processing 
module  in  which  a  biomimetic  image  processor 
detects  road  edges  in  real-time,  and  a  Driver’s  Vision 
Enhancer  (DVE)  which  displays  the  road  image, 
detected  boundaries  and  road-side  terrain  steepness 
in  real-time  for  the  driver. 

1.  INTRODUCTION 

A  crew  of  two  soldiers  is  in  a  High-Mobility 
Multipurpose  Wheeled  Vehicle  (HMMWV)  driving  at 
night  without  the  headlights  turned  on,  on  an 
unpaved,  unmarked  road.  Unexpectedly,  a  camel 
crosses  the  road  and  the  driver  quickly  swerves  to 
the  side  of  the  road  to  avoid  hitting  the  camel. 
Unbeknownst  to  the  driver  and  the  passenger,  there 
was  a  twelve  foot  ditch  on  the  passenger  side  of  the 
vehicle.  The  HWWVW  rolled  over  and  ejected  the 
soldiers  out  of  their  seats,  since  neither  of  them  were 
wearing  seatbelts.  Neither  of  the  soldiers  were 
seriously  injured,  but  they  did  suffer  a  broken  arm 
and  a  sprained  neck  This  accident  is  only  a 
theoretical  example  of  what  soldiers  can  experience 
while  driving  military  vehicles  in  various  theaters.  A 
recent  report  identified  that  one  cause  of  rollovers  is 
the  driver’s  inability  to  assess  rollover  threat,  such  as 
a  cliff,  soft  ground,  water,  or  culvert  on  the 
passenger  side  of  the  vehicle.  The  vehicle’s  width 
hinders  the  driver’s  field  of  view.  Each  year,  Army 
personnel  and  soldiers  are  involved  in  military 


vehicle  accidents,  many  of  which  could  have  been 
avoided.  There  are  several  ways  to  decrease  military 
vehicle  rollovers  and  associated  accident  fatalities; 
one  way  is  a  road  edge  detection  and  driver  warning 
system.  Engineers  and  research  scientists  are 
currently  developing  a  road  edge  detection  and 
driver  warning  system  to  warn  the  driver  of  potential 
rollover  threats  and  to  keep  the  driver  from  veering 
off  the  side  of  the  road. 


2.  ROAD  EDGE  DETECTION  SYSTEM 


Fig.  1 :  General  architecture  of  the 
road  edge  detection  system 

Figure  1  shows  the  conceptual  general 
architecture  of  the  system.  This  diagram  shows  the 
video  and  LWIR  cameras  which  provide  images  for 
daytime,  nighttime,  smoke,  fog,  snow,  and  rain 
operation.  The  laser  scanner  measures  the  terrain 
steepness.  The  video  images  are  read  in  by  a 
processing  unit.  The  algorithms  embedded  in  the 
processing  unit  include  fly  vision-based,  edge  fusion, 
road  edge  identification  and  roll-over  threat 
determination.  The  fly  vision-based  algorithm  is 
applied  to  detect  all  edges  present  in  the  video  and 
LWIR  images.  The  edge  fusion  algorithm  combines 
the  detected  edges  into  a  single  set  of  edges.  It  is 
part  of  a  processing  unit  that  fuses  images  from  the 
cameras,  extracts  edges  from  the  images,  detects 
road  edges  and  identifies  the  road  boundary  on  the 
passenger  side  of  the  vehicle,  and  integrates  the 
terrain  steepness  information  with  the  road  boundary 
information  to  assess  rollover  threat  and  warn  the 
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driver.  The  road  edge  identification  algorithm 
identifies,  among  all  the  detected  edges,  those 
pertaining  to  the  road  boundary.  The  rollover  threat 
detection  algorithm  detects  the  presence  of  negative 
roadside  terrain  on  the  passenger  side  of  the  road, 
and  warns  the  driver  acoustically  and  visually  using 
alarm  signals. 


Fig.  2:  Video  and  LWIR  Cameras  potential 
Mounting  Positions. 


Fig.  3:  Potential  Scanning  Range  of  the 
Camera  System. 

Figure  2  shows  an  example  of  how  the 
monochrome  video  and  LWIR  cameras  and  laser 
scanner  will  be  mounted.  The  different  shading  in 
Figure  3  indicates  the  wider  field-of-view  that  the 
camera  system  has  versus  the  laser  scanner.  The 
camera  system  provides  road  boundary  detection 
and  laser  scanner  provides  range  and  roadside 
terrain  steepness  information. 

2.1  Video  Camera 

In  the  past,  video  images  have  been  used  to 
detect  and  track  road  boundaries  and  lane  markers 
(Wang  et  al.,  2002;  Want  et  al.,  2004).  The  images 


obtained  from  the  video  provide  the  best  results  on 
roads  with  land  markers  when  there  is  daylight  and 
good  weather  conditions.  Video  images  are  difficult 
to  use  during  nighttime,  rain,  snow,  fog,  smoke  and 
on  unmarked  terrain.  To  improve  viewing  the 
scenery  during  nighttime,  or  in  the  presence  of 
smoke,  fog,  snow,  and  rain,  and  on  all  terrain 
conditions,  both  video  and  LWIR  cameras  are 
needed. 

LWIR  cameras  are  not  affected  by  bright 
sunlight  or  headlights,  but  are  sensitive  to 
temperature.  There  are  two  types  of  infrared  imagery 
technologies,  cooled  and  uncooled  infrared 
detectors.  The  cooled  infrared  also  has  an  integrated 
thermo-electric  cooler  (TEC),  which  makes  the  unit 
heavy  and  oversized.  Furthermore,  it  also  consumes 
more  power.  In  addition,  the  cooled  infrared 
cameras  deliver  higher  resolution  images,  since  the 
cameras  often  use  Barium  Strontium  Titanate 
ferroelectric,  but  they  are  bulky  and  expensive,  each 
unit  costs  more  than  $25,000,  which  is  a  hefty  price 
tag  in  today’s  budget-minded  Army.  The  uncooled 
infrared  camera,  on  the  other  hand,  is  more  compact 
since  each  unit  uses  a  micro-bolometer  that  is 
sensitive  to  radiation  in  the  7-14  micron  band  for 
night  time  use.  The  uncooled  infrared  camera  is 
more  affordable,  each  unit  costs  less  than  $10,000. 

The  image  display  from  a  LWIR  camera  enables 
the  driver  to  distinguish  the  different  types  of 
materials  used  on  the  roads,  such  as  asphalt,  dirt,  or 
gravel,  and  from  the  embankment,  such  as  grass, 
rock  or  a  cliff.  This  distinction  provides  better 
contrast,  thus  making  detection  of  the  road  boundary 
and  embankment  easier  during  nighttime  driving  and 
under  different  weather  conditions.  However,  utilizing 
both  the  monochrome  and  LWIR  cameras  does  not 
supply  sufficient  images  for  detecting  terrain 
depression  as  accurately,  since  the  cameras  do  not 
provide  depth  perception.  To  overcome  this  hurdle,  a 
laser  scanner  is  also  needed. 

2.2  Laser  Scanner 

The  laser  scanner  provides  depth,  or  steepness, 
perception  since  the  laser  scanner  can  scan  any 
point  in  space  to  indicate  the  present  of  a  cliff  or  an 
obstacle.  An  increase  in  the  slope  of  the  scanning 
denotes  a  terrain  depression  such  as  a  cliff  that  is  on 
the  passenger  side,  while  a  decrease  in  the  slope  will 
denote  positive  terrain  such  as  a  big  rock  or  an 
obstacle  that  is  on  the  passenger  side  of  the  vehicle. 

The  laser  scanner  that  the  researchers  and 
engineers  are  considering  has  a  light  beam  that  is 
invisible  to  the  naked  eye  and  should  not  interfere 
with  the  LWIR  imaging.  The  laser  light  beam  should 
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operate  in  near  infrared,  or  less  than  1pm 
wavelength,  while  the  LWIR  is  sensitive  only  to  long 
wavelengths  greater  than  7.0pm.  The  laser  light 
needs  to  be  inconspicuous  and  should  not  signal  the 
vehicle’s  presence  to  an  enemy.  The  laser  scanner 
needs  to  have  at  least  a  resolution  of  10mm,  a  90 
degree  angle  between  the  incident  beam  and  the 
reflecting  material  for  proper  measurement,  and  at 
least  a  75m  detection  range.  A  75m  range  is  needed 
because  a  short  distance  detection  range  is  not 
beneficial  to  the  driver.  With  longer  distance  range, 
the  system  can  detect  steepness  of  the  terrain  far 
enough  ahead  of  the  vehicle  to  warn  the  driver  and 
allow  the  driver  enough  time  to  react  to  avoid  the 
obstacle  or  the  rollover  threat. 

3.  PROCESSING  SYSTEM 

After  the  monochrome  and  LWIR  cameras 
capture  the  images,  algorithms  are  needed  to 
process  the  images.  These  algorithms  are  the  fly 
eye-based  algorithm  to  detect  road  edges,  an 
algorithm  to  fuse  monochrome  video  and  LWIR 
images,  a  road  boundary  identification  algorithm,  and 
a  roll-over  threat  detection  algorithm. 

3.1  Road  Edge  Detection  Algorithm 

The  road  edge  detection  algorithm  extracts  the 
edges  in  the  images.  Typically,  images  from  the 
video  require  intensive  computational  power  and 
time.  This  lag  causes  a  delay  of  the  information 
displayed  to  the  user  so  it  is  not  in  real-time.  To 
address  this,  an  innovative  method  to  process  the 
video  images  has  been  developed,  which  requires 
less  computational  power  and  provides  real-time 
information  to  the  user.  This  technique  applies  the  fly 
eye-based  vision.  This  concept  of  the  fly  eye-based 
vision,  or  biomimetric  edge  vision,  is  based  on  the 
neurobiology  of  insect  vision,  since  insects  rely  on 
edge  detection  to  avoid  obstacles  (Agassounon  et 
al.,  2004;  Douglass  and  Strausfseld,  2003; 
Sinakevitch  and  Strausfeld,  2004).  The  way  an 
insect  uses  its  vision  significantly  reduces  the 
intensity  of  visual  signal  processing  in  the  insect’s 
brain. 

An  example  of  using  the  fly  eye-based  edge 
detection  algorithm  is  shown  in  Figure  4a  and  4b. 
Figure  4a  shows  an  image  of  the  road  and  Figure  4b 
shows  the  same  images  using  the  fly-eye-based 
detection  algorithm.  From  looking  at  Figure  4b,  one 
notices  that  the  algorithm  captures  the  relevant 
contour  pixels  of  the  image,  which  are  the  lane 
maker,  cliff  edge  and  road  boundary.  The  double 
edge  lines  on  the  passenger  side  correspond  to  the 


lane  marker  and  the  actual  road  edge  where  the  cliff 
begins. 
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Fig.  4:  (a)  Image  of  the  Road  and  (b)  Image 

Using  Fly-eye  Based  Detection  Algorithm 

Since  an  insect  only  sees  the  edges,  software 
leveraging  this  type  of  cognitive  vision  response 
system  increases  the  robustness  and  computational 
speed  of  the  system.  The  fly-eye-based,  edge- 
detection  algorithm  is  thousands  of  times  faster  than 
the  traditional,  gradient-based  algorithm  because 
only  relevant  edge  data  is  utilized.  This  process  also 
requires  less  storage  space  because  only  the 
relevant  edge  information  is  captured.  The  other 
information  is  not  necessary. 

The  monochrome  video  and  LWIR  images 
obtained  from  the  ultra-fast,  fly-eye-based  edge- 
detection  algorithm  need  to  be  fused  to  form  a 
single,  robust  estimate  of  the  detected  edges.  This 
process  is  used  to  increase  the  probability  of 
detection  of  the  road  edge  and  to  decrease  the 
probability  of  a  false  alarm. 

3.2  Fusion  Algorithm 

Using  only  the  video  camera  or  the  LWIR 
camera  does  not  optimize  the  detection  of  the  road 
edge.  Thus,  both  the  video  and  the  LWIR  cameras 
need  to  be  used  especially  for  driving  at  night  or 
under  different  weather  conditions.  The  video 
camera  and  the  LWIR  camera  will  be  placed  and 
oriented  such  that  both  the  images  share  roughly  the 
same  field-of-view,  so  that  the  images  captured  by 
both  cameras  can  be  superimposed.  Therefore,  the 
image  from  each  camera,  once  processed  through 
the  edge  detector,  will  provide  a  set  of  detected 
edges  representing  the  edge  information. 

The  fusion  process  merges  the  edges  detected 
in  each  of  the  individual  images  captured  by  the 
video  and  LWIR,  into  a  single,  enhanced  set  of 
edges  that  can  be  displayed  to  the  driver  on  a 
monitor.  This  method  is  modular  in  the  sense  that 
one  detector,  either  the  video  or  LWIR  cameras,  can 
be  removed  for  a  given  platform  or  application 
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without  changing  the  fundamental  algorithm 
operation.  This  allows  the  framework  to  be  applied  to 
a  wide  range  of  systems  of  various  practicality  and 
payload. 


Fig.  5:  Edges  fusion  example.  (Images  are 
courtesy  of  Zeiss  Optronics.) 

The  fusion  algorithm  uses  a  weighted  sum  of  the 
pixel  intensities  of  detected  edges  in  the  visible  and 
LWIR  images.  Thus,  this  method  integrates  the  edge 
information  into  a  single-edge  image.  Equation  1 
shows  how  the  weighted  sum  is  calculated: 

Ifused  (X,y)—  a  I  visible  (X,y)  Ct)  l|_WIR(X,y)  (1) 

In  Equation  1,  lvisib|e  (x,y)  and  l|_wiR(x,y)  are  the 
intensities  of  the  edge  pixels  in  the  visible  and  the 

LWIR  images  at  position  (x,y),  and  a  is  the  weight  of 
the  fusion  (0  <  a<1 ).  The  contribution  from  either  the 
LWIR  camera  or  the  video  camera  can  be  increased 
or  decreased  by  varying  a.  Figure  5  shows  and 
example  of  the  edge  fusion,  where  a=0.5.  In  Figure 
5,  the  picture  on  the  left  is  of  the  road  taken  at  night 
using  a  monochrome  video  camera,  while  the  picture 
on  the  right  is  with  a  LWIR  camera.  The  bottom 
picture  is  a  superimposed  image  of  the  two  pictures 
using  the  fusion  algorithm. 


3.3  Road  Boundary  Identification  Algorithm 

The  road  boundary  identification  algorithm 
identifies  edges  that  are  part  of  the  road  boundary 
and  reconstructs  the  road  boundary  with  the  selected 
edges  since  not  all  the  edges  are  associated  with  the 
road  boundary.  The  algorithm  applies  a  model-based 
technique  to  fit  consecutive  segments  of  the  road 
boundary  with  a  curve  fragment  by  fitting  the  road 
edge  segment  with  a  short  2D  vector.  This  algorithm 
is  similar  to  the  one  proposed  by  Wilson  (Wilson  and 
Dickson,  1999). 

This  algorithm  works  by  automatically  searching 
for  the  origin  of  the  road  boundary  in  the  lower  right 
quadrant  of  the  edges  image  by  looking  for  the  origin 
of  the  bottommost,  rightmost  straight  line.  Then  a 
short  vector  of  selected  length  or  fitting  window  is 
drawn  from  the  origin.  The  algorithm  then  counts  the 
number  of  detected  edges  belonging  to  the  vector  by 
rotating  the  vector  about  the  origin  point  within  a 
region  defined  by  the  original  reference  to  find  the 
best  fit.  The  best  fit  corresponds  to  the  position  for 
which  the  fitting  vector  has  the  most  edge  points  and 
represents  the  detected  fragment  for  the  road 
boundary.  The  tip  of  the  vector  becomes  the  new 
origin.  The  procedure  is  repeated  to  detect  the  full 
road  boundary  fragment  by  fragment.  Figure  6 
illustrates  the  road  boundary  identification  procedure. 


Fig. 6:  Illustration  of  road  boundary  identification 

procedure;  (a)  a  fitting  vector  is  drawn  from  a 
selected  origin,  rotated  about  the  origin  to  find 
the  best  fit;  (b)  tip  of  vector  becomes  the  new 
origin  and  the  procedure  is  repeated  to  detect 
the  full  road  boundary  fragment  by  fragment. 

3.4  Rollover  Threat  Detection  Algorithm 

Once  the  road  boundary  is  detected,  the 
presence  of  a  negative  terrain  on  the  roadside,  and 
whether  the  vehicle  is  on  course  to  cross  the 
boundary  on  the  passenger  side  and  rollover  are 
also  considered.  In  addition,  the  time  to  road  edge 
crossing  is  computed.  To  estimate  the  road  edge 
crossing,  the  threat  detection  algorithm  uses  the 
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vehicle’s  heading,  current  speed  of  the  vehicle,  and 
the  distance  to  the  point  of  rollover.  When  a  terrain 
depression  is  detected  on  the  passenger  side  or 
when  the  road  edge  crossing  is  below  a  certain 
threshold,  which  is  related  to  the  driver  reaction  time, 
an  acoustic  signal  and  a  visual  signal  will 
simultaneously  warn  the  driver  of  the  danger.  The 
acoustic  signal  could  be  something  similar  to  the 
stall-horn  on  small  airplanes  or  a  beeping  sound,  and 
the  visual  signal  could  be  a  red  flashing  light-emitting 
diode  (LED)  on  the  Graphical  User  Interface  (GUI) 
screen  to  indicate  rollover  point.  The  threshold  will 
be  set  according  to  the  driver’s  reaction  time,  which 
will  allow  the  driver  to  steer  the  vehicle  back  on 
track.  The  output  of  the  processing  system  will  be 
displayed  on  a  screen  inside  the  vehicle  using  a  GUI. 

3.5  Driver’s  Vision  Enhancer  (DVE) 

The  information  obtained  from  the  LWIR  and 
monochrome  video  camera  system  is  combined  with 
the  digital  signal  processing  unit  to  be  displayed  on 
the  Driver-Vehicle  Interface  (DVI).  The  Driver- 
Vehicle  Interface  is  similar  to  a  GUI.  The  DVI  allows 
the  driver  to  interact  with  and  control  the  road  edge 
detection  system.  The  DVI  will  be  used  to  display  the 
current  image  of  the  terrain,  such  as  the  road  ahead 
as  seen  from  the  monochrome  and  LWIR  cameras, 
the  detected  road  edges,  and  the  roadside  terrain 
steepness  information.  The  displayed  image  will  be  a 
composite  image  made  of  the  images  captured  by 
the  video  and  LWIR  cameras.  The  driver  will  be  able 
to  initialize  the  edge  detection  and  identification 
procedure  by  selecting  a  point  on  the  screen.  This 
point  will  be  used  as  the  starting  point  for  the 
algorithm.  Other  features  that  will  be  considered  for 
the  display  are  GPS,  compass,  and  dead  reckoning. 
Figure  7  shows  an  example  of  a  potential  DVI 
display  for  the  driver. 
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Fig. 7:  Conceptual  design  for  the  DVI  Display 


4.  CONCLUSION 

The  development  of  the  road  edge  detection  and 
driver  warning  system  will  decrease  the  number  of 
rollover  incidents  since  the  system  will  notify  the 
driver  of  potential  rollover  threats.  This  detection 
system  utilizes  a  unique,  ultra-fast,  image-processing 
algorithm  based  on  the  neurobiology  of  insect  vision, 
specifically  the  vision  of  a  fly.  The  system  consists  of 
a  Long-Wavelength  Infrared  (LWIR)  camera  and 
monochrome  video  camera  system,  a  long-range 
laser  scanner,  a  processing  module  in  which  a 
biomimetic  image  processor  detects  edges  in  real¬ 
time,  and  a  Driver-Vehicle  Interface  (DVI)  which 
displays  the  road  image,  detected  boundaries  and 
road-side  terrain  steepness  in  real-time  for  the 
driver. 

The  rapid,  road-edge  detection  based  on 
biomimetic  image  processing  has  the  potential  to  be 
implemented  in  the  Future  Combat  Systems’  (FCS) 
Automated  Navigation  System  (ANS).  The  road  edge 
detection  system  can  enhance  the  capability  of  the 
FCS  by  aiding  mobile,  autonomous  ground  robots, 
detecting  potential  rollover  threats,  and  preventing 
the  autonomous  vehicle  from  rolling  over. 
Furthermore,  this  system  can  be  embedded  in  FCS 
Manned  Ground  Vehicle  (MGV)  to  reduce  rollover 
accidents,  thus  minimizing  the  number  of  injuries 
and  ultimately  saving  soldiers’  lives. 
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